Bottom-up or Top-down? Dynamics of Deep Representations via Canonical-correlation Analysis
نویسندگان
چکیده
We present a versatile quantitative framework for comparing representations in deep neural networks, based on Canonical Correlation Analysis, and use it to analyze the dynamics of representation learning during the training process of deep networks. We find that layers converge to their final representation from the bottom-up, but that the representations themselves migrate downwards in the network over the course of learning.
منابع مشابه
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Understanding and Improvement
We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality...
متن کاملEvent-Related Potentials of Bottom-Up and Top-Down Processing of Emotional Faces
Introduction: Emotional stimulus is processed automatically in a bottom-up way or can be processed voluntarily in a top-down way. Imaging studies have indicated that bottom-up and top-down processing are mediated through different neural systems. However, temporal differentiation of top-down versus bottom-up processing of facial emotional expressions has remained to be clarified. The present st...
متن کاملA Comparative Study of Effect of Bottom-up and Top-down Instructional Approaches on EFL Learners’ Vocabulary Recall and Retention
This quasi-experimental study investigated the effect of bottom-up and top-down instructional approaches on English as a foreign language (EFL) vocabulary recall and retention. To this end, 44 high school students from two intact classes were assigned to bottom-up (n = 21) and top-down (n = 23) groups. The participants were exposed to 20 hours of explicit vocabulary instruction during 10 weeks ...
متن کاملEffect of Metacognitive Strategy Training and Perfectionism on Listening Comprehension Sub-Processes
The present study aimed to examine any possible relevance of perfectionism as a personal trait variable, in moderating the effectiveness of meta-cognitive instruction on bottom-up and top-down sub-processes of listening comprehension with a sample of EFL learners in Iranian context. To this end, 94 female EFL learners were selected from among 136 EFL learners at Andisheh Language Institute in M...
متن کاملDeep Predictive Coding Network for Object Recognition
Inspired by predictive coding in neuroscience, we designed a bi-directional and recurrent neural net, namely deep predictive coding networks (PCN). It uses convolutional layers in both feedforward and feedback networks, and recurrent connections within each layer. Feedback connections from a higher layer carry the prediction of its lower-layer representation; feedforward connections carry the p...
متن کامل